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Overview 

• 31 researchers (14 Ph.D.’s) 

• 20+ summer interns yearly 

• 75% NASA work 
(HEOMD, STMD, SMD) 

• 25% reimbursable (Google, etc.) 

• SBIR / STTR (10 current proj.) 

Research themes 

• Automated planetary mapping 

■ Base maps & terrain models 

■ Geospatial data systems 

• Robots for human explorers 

■ Improve efficiency & productivity 

■ Pre-cursor & “follow-up” work 

• Public service 

■ Disaster response & outreach 
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(IRG) 



irg.arc.nasa.gov 
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IRG Collaborations (2010-2013) 


Academic 


Commercial 



ROBOTICS 

INSTITUTE 


Mlir 

Massachusetts 
Institute of 
Technology 



Arizona Stati 
University 


Google Research 



Stottler Henke 


University of Idaho 



Eidgenossische Technische Hochschule Zurich 
Swiss Federal Institute of Technology Zurich 



SETI Institute 


KAIST 

Korea Advanced Institute of Science and Technology 
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Smarter Software Solutions 

IENERGID 

HONEYBEE robotics 



Optech , 



PI 


TRAB 


DO 

CD 


PRCI 


Protolnnovations 


Government 
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Robotics for Human Exploration 


Purpose 

• Increase human productivity 

• Improve mission planning & execution 

• Transfer some tasks to robots 
(tedious, repetitive, long-duration) 

Before Crew 

• Recon (scouting) & prospecting 

• Site prep, deploy equipment, etc. 

Supporting Crew 

• Inspection, mobile camera, etc. 

• Heavy transport & mobility 


After Crew 


• Follow-up & close-out work 

• Site survey, supplementary tasks, etc. 
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Robots 






Perception 









User Interfaces 
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Google Earth Ops (2008 - 2011) 
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K10 Data Browser (2010 - 2011) 



Flight Control Team 


Flight Director 
(“Flight”) 


Data Downlink 
(“Data”) 


— Robot Systems Lead 


Robot Support Team 


Science Operations Team 


Interactive Ground Control (2008 - 2010) 







Architecture 



RoverSW (2006 -2011) - 


imp MangMnn Klfl CmlrnllM J 




Mission Management Team 
Tactical Science Ops 
Strategic Science Ops 


Earth 





xGDS: Exploration Ground Data System 
(2009-2011) 
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RAPID (2009-2011) 


/ 


\ 



RAPID 
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Middleware 
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K10 Robot at Haughton Crater, Canada 
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Mars in Google Eart 


Explore Mars in 3D 

• Released Feb. 2, 2009 

• Co-developed with Google 

• NASA Ames created content 
& processing scripts 

Content 

• Global maps: topography, 
infrared, historical, etc. 

• Imager footprints & overlays 
(HiRISE, CTX, MOC, ...) 

• Mars rover tracks & 
color panoramas 

• Tours (Bill Nye & Ira Flatow) 

• Live from Mars: THEMIS 

• And much more ... 
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Moon in Google Eart 


Explore the Moon in 3D 

• Released July 20, 2009 

• Co-developed with Google 

• NASA Ames created content 
& processing scripts 

Content 

• Global maps: topography, 
geologic, historical, etc. 

• Spacecraft imagery: Apollo, 
Lunar Orbiter, etc. 

• 3D models of spacecraft, 
landers, and crew rovers. 


• Tours (Andy Chaikin, Buzz 
Aldrin & Jack Schmidt) 


And much more ... 
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World WideT elescope 


Complete HiRISE Mosaic 

• Mars Reconaissance Orbiter 
HiRISE imager 

• 74,000 images 

• Each image: 20K x 50K pixels 
(> 1 GB / image) 


Mosaic stats 


Tile Dimensions 

256 x 256 pixels 

Root Tiles / Image 

15,000 

Tile Space 

25 KB 

Tiles Total 

229 million 

Total Mosaic Size 

5.7 TB 
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Mars 




Microsoft* Research 


Worldwide Telescope 

^ A 

Welcome to WWT | Mars 
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Exploration Ground Data System (xGDS) 


xGDS is ... 

• Map content management 

• Planning tool 

• Real-time plots, maps, notes 

• Post-processing data archive 

• Browse and search tools 

Users 

• Field scientists 

• Planetary scientists 

• Mission planners 

• Flight controllers 

• Local & distributed teams 
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xGDS Architecture 
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IRG Open Source Software 


Vision 

Workbench 




Exploration 
Ground 
Data Systems 

(xGDS) 


RoverSW 




Visual 

Environment 
for Remote 
Virtual 
Exploration 

(VERVE) 


Neo Geography 
Toolkit 

(with Ames 
Stereo Pipeline 




RAPID 

(NASA robot 
middleware) 
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Questions? 
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Intelligent Systems Division 
NASA Ames Research Center 

irg.arc.nasa.gov 
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NASA Ames Planetary Mapping 

Making NASA’s data more rapidly and universally available 



h tt p :// i rg. a r c . n as .go v 
Intelligent Robotics Group 
NASA Ames Research Center 


Contact: terry.fong@nasa.gov 



Really Big Data 

Why do we need to build automated systems to handle large volumes of Planetary data? 


We are in the midst of a major geospatial information explosion. For the first 
time, I0‘s of Terabytes of map data are being collected by robotic explorers. 



MGS Odyssey MER MRO Apollo SELENE ChangE-1 Chandra LRO 

yaan-1 


Approximate data 
volumes from 
various Mars (red) 
and Lunar (blue) 
missions. 

Data volumes are 
in Terabytes! 

Source: B. A. Archinal, L. 
R. Gaddis, R. L. Kirk, T. M. 
Hare, and M. R. Rosiek. 
Urgent Processing and 
Geodetic Control of Lunar 
Data . Workshop on 
Science Associated with 
the Lunar Exploration 
Architecture, 2007. 
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Really Big Images 

Why do we need to build automated systems to handle large volumes of Planetary data? 


Apollo Metric Camera 
(16,000x16,000) 


HRSC 

(5184x 

16000) 


CTX 

(5064x 

16000) 


MOC-NA 
(2048x4800)| 


MER □ 
(1024x1024) 


HiRISE 

(20,000x40,000) 


Nominal Resolutions for Various Imagers. All sizes given in pixels. 
Apollo Panoramic Camera film scans are not shown (25400 x 244000 pixels)! 


LROC 

(10000x50000) 


In the past, widely used maps 
such as the Viking MDIM mosaic 
took years to produce. 



Today, human intensive 
processes can be automated 
so that data can be processed 
& distributed more rapidly. 
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Some of our Projects 

Automated 3D surface reconstruction, mosaicking, gigapixel imaging 


0 Bridging the gap between PDS and 
Geo-browser platforms 

Enabling faster, easier, and universal access 

0 Developers in the Lunar Mapping 
and Modeling Program (LMMP) 

Providing stereo-derived topography and imagery 
from Apollo Metric Camera scans 

0 LROC & HiRISETeam Members 

Validating and estimating the errors of 
LROC-derived DTMs 

0 Assorted other projects: 

Vision Workbench, Ames Stereo Pipeline, Neo- 
Geography Toolkit, Small body mapping 



Automated Stereo Reconstruction of Jackson Crater (LROC) 



Human and Robotic Mission 
to Small Bodies: 

Mapping, Planning and Exploration 

A Study for the Advanced Exploration Systems 
(AES) Joint Robotic Precursor Activities (JRPA) 
Project 

Are V. Nefian 

Carnegie Mellon University, Moffett Field. CalUomia 
Juke Belief ose 

Carnegie Mellon University. Moffett Field. CalUomia 
Ross A. Beyer 

SET1 Institute. Mountain View. CalUomia 
Laurence Edwards 

Ames Research Center. Moffett Field. California 
Pascal C. Lee 

Mars Institute. Moffett Field. California 
Anthony Cotaprete 

Ames Research Center. Moffett Field. California 
Terrence Fong 

Ames Research Center. Moffett Field, California 
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Our Automated Pipeline 

Unified architecture for processing & serving planetary data through web services 


•Our geospatial data 
pipeline has been 
developed to 
automatically produce 
high-quality planetary 
maps and models 

•It is extremely flexible 
& extensible, 
supporting many data 
source and open 
standards & protocols. 

•Our software stack 
runs on Pleiades, the 
NASA Ames 
supercomputer 
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Ames Stereo Pipeline 

Open-source automated stereogrammetry software 




Models from 
traditional stereo 
cameras using the 
TSAI format. 


Models of the other 
planets using 
USGS’s ISIS cube 
files. 


Models of Earth 
using Digital Globe 

Stereo 1 B imagery. 


Models of Earth 
using imagery that 
contains an RPC 
Model. 
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Ames Stereo Pipeline (ASP) 

Open-source automated stereogrammetry software 


• What it is... 

• Command-line tools for computing 
clusters and super computers 

• C++ code hosted on GitHub 

• Binaries available for Linux and OS-X 

• Apache 2 license 

• Data processed with ASP 

• Apollo Metric Camera 

• Digital Globe I B products 

• Lunar Reconnaissance Orbital 
Camera (LROC-NA) 

• Mars High Resolution Imaging Science 
Experiment (HiRISE) 

• Mars Orbiter Camera (MOC) 

• MRO Context Camera (CTX) 


NATIONAL AERONAUTICS 
AND SPACE ADMINISTRATION 


♦ABOUT NASA I ♦LATEST NEWS I ♦MULTIMEDIA I ♦MISSIONS I ♦ MY NASA I ♦WORK FOR NASA 


Stereo Pipeline 


• Stereo Pipeline Examples 


w 



I The Stereo Pipeline 


I The NASA Ames Stereo Pipeline (ASP) is a suite o 
I geodesy 8 stereogrammetry tools designed for pro 
I planetary imagery captured from orbiting and land< 
| explorers on other planets. It was designed to proc 
imagery captured by NASA spacecraft and product 
products including digital elevation models (DEMs 
| projected imagery, and 3D models. These data pro 
tollable for science analysis, mission planning, am 
outreach 


The SI 


io Pipeline is part of the NASA NeoGeogr 


Search 

Intelligent Systems Division 


This version of the Stereo Pipeline is designed as 
an existing installation of the USGS Integrated Soft 
Imagers and Spectrometers (ISIS). ISIS is widely uf 
planetary science community tor processing raw sj 
imagery into high level data products of scientific 
map projected and mosaicked imagery. 

Downloading and Installation 

The latest Stereo Pipeline binary distnbutron is ava 
Version 2.2.2 binanes are designed to be compatit 
v3.4,3. Please install (SIS if you wish to process N4 
terrestrial imagery. Users wishing to processing Dii 
GeoEye, or perspective imagery do not need to dor 
anything else 

• Binaries 

u Stereo Pipeline v2.2J2 (ISIS v3.4.3 op 

• Mac OS X 106+ 

• Linux-64bit 

• Linux-32bit 

. Linux-64bit tor GLIBC2.1 7 and 


The Ames Stereo Pipeline: 

NASA’s Open Source Automated Stereogrammetry Software 
A part of the NASA NeoGeography Toolkit 
Version 2.2.0 


tiny.cc/ames-stereo-pipeline 
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Apollo Zone DEM 

High-resolution terrain model (digital elevation map) 

• Mosaick of 4,000 images 

• Apollo Metric Camera 

• 73,728 x 368,640 pixels 

• Equatorial Lunar Surface 

(38S-34N lat) 

• 1 ,024 pixel / deg 

• Vert, acc 40.9m (LOLA) 

• Vert, stdv 37.8m 

• Horiz. acc 9 1 .3m (LOLA) 

• Controlled to LOLA 
through LRO-WAC 

• 40,000 CPU hours 
(4 days on NASA Pleiades) 


NASA 
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Albedo Reconstruction 

Scalability from single core to super computer (GNU parallel) 


• I mage formation model: 

Camera Transfer Function, Albedo, Exposure 
Estimation, Surface Reflectance, and Shadow 


Camera 

position 


Orbital Image, DEM 

I 


•For the Moon, uses Lunar-Lambertian Model 
(can be extended to non-Lunar surfaces) to 
reduce the effect of varying illumination 

•Overlapping images allows for shadow removal 


Ortho-projection 

I 

Shadow Map, Weights 

I 

Init Exposure 



Original Reconstructed albedo 


Init Albedo 

I 

Update Exposure 

l 


Update Albedo 



Sun 

p osition 
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Albedo Reconstruction 

Hadley Rille (Apollo Metric Camera images) 



Original 
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Reconstructed albedo 


NASA 




Albedo Reconstruction 

Hadley Rille (Apollo Metric Camera images) 
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Apollo Zone DIM 


High-resolution base map (digital image mosaick) 



Albedo Maps 


Data 

Resolution 

Coverage 

LRO-WAC 
1 LRO-NAC 
Apollo Metric 

1 00 m/pixel 
1 m/pixel 
10 m/pixel 

100% 

<2% (early 20 13) 
18% 
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Mosaicking & Blending 

Highly accurate reconstruction, blending, and error modeling 
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ASP for Earth 

Cryospheric mapping for Earth science ( glaciology , climate change , etc) 


Jakobshavn Glacier, Greenland 

• 6 input images 

• 2500 km 2 coverage 

• 5 m/post 
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ASP for Earth 

Non-vegetated areas 
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ASP for Earth 

60% success rate processing Digital Globe stereo pairs without human input 
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• Colorized DEM of the 
Basalt Hills Quarry 

g • Used for planetary rover 
analog testing 

• Based on DigitalGlobe 
WorldView imagery 
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SRTM DEM (~31 Meters / Pixel ) USGS NED DEM (9 Meters / Pixel ) ASP WV 1 DEM (0,5 Meters /Pixel) 
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NASA / Google 

Explore the Moon and Mars in 3D 



• “Google Moon’’ & “Google Mars’’ 
provide data availability & fusion for 
planetary data 

• Includes “live’’ imagery of Mars 
from the THEMIS camera (appears 
on-line 2-4 days after downlink) 

• Guided tours of the Moon and 
Mars narrated by Buzz Aldrin, Jack 
Schmitt, Ira Flatow, and Bill Nye. 

Try it for 
yourself in 
Google 
Earth 5.0! 




Modern / Historical Base Maps 



Geospatial Image 
Browsing/Indexing 
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Geologic Maps 



High Resolution 3D Terrain 



Tours Narrated by Notable 
Scientists and Astronauts 



Geo-located 
Panoramic Imagery 


NASA 
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This work was done at ARC under a reimbursible space act agreement (RSAA). 
The RSAA does not imply exclusive access to NASA data or to our team. 
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Google Moon 

Released July 20, 2009 



• “Moon for Google Earth” 

• Co-developed with Google 

• IRG created content, processing 
scripts, and base maps 

• Built in to Google Earth 5.0 

• Content 

• Global maps: topography, geologic, 
historical, etc. 

• Spacecraft imagery:Apollo, 
Clementine, Lunar Orbiter 

• 3D models of spacecraft, landers, 
and crew rovers. 

• Tours (Andy Chaikin, Buzz Aldrin 
and Jack Schmidt) 

• And much more ... 



Letronne Crater: Plunging Into the Ocean ol Storms 


NASA 
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Google Mars 

Released February 2,2009 



• “Mars for Google Earth” 

• Co-developed with Google 

• IRG created content, processing 
scripts, and base maps 

• Built in to Google Earth 5.0 

• Content 

• Global maps: topography, infrared, 
historical, etc. 

• Imager footprints & overlays: 
HiRISE, CTX, MOC, etc. 

• MER tracks & panoramas 

• Tours (Bill Nye & Ira Flatow) 

• Live from Mars:THEMIS images 
within hours 

• And much more ... 
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NASA / Microsoft 

Bringing the Mars experience to World Wide Telescope 


WWT I Mars 


w 


• Featuring the largest digital image 
mosaic of Mars ever created 


• Data sets for WWT Mars were 
created using the NASA Nebula 
cloud computer 


NEBULA Cloud Computing Platform 


• Guided tours of Mars narrated by 
Dr. Carol Stoker and Dr. Jim Garvin 

http://worldwidetelescope.org 
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MOC HiRISE 


Total # of images 

74,359 

1 3,342 

Pixels / Image 

1 6 Megapixels 

1 .25 Gigapixels 


Total Image Tiles 

~38 Million 

-526 Million 

Total Mosaic Size 

843 Gigabytes 

1 2 Terabytes 
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This work was done at ARC under a reimbursible space act agreement (RSAA). 
The RSAA does not imply exclusive access to NASA data or to our team. 
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WWT Mars Base Layers 

New global maps and historical imagery 


Color MDIM v2 
(merged w/ MOC- 
WAC mosaic) 



Mars Topography 
(MOLA) 


/ 



Nathaniel Green 


(1877) 


Historical Imagery 



Giovanni Schiaparelli 
(1890) 



Percival Lowell 
(1896) 



Eugene Antoniadi MEC-I Prototype (USAF) 
(1909) (1962) 


NASA 
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WWT High Resolution Layers 

From Mars Global Surveyor and Mars Reconnaissance Orbiter 


74,359 Mars Global Surveyor MOC Images 



1 3,342 Mars Reconnaissance Orbiter HiRISE Observations 
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Access to Planetary Data 

Providing 2D and 3D NASA imagery to cutting-edge geo-browser platforms 


• NASA has done an exemplary job of 
archiving its data and making it 
publicly available (e.g. the PDS and 
DAACs), but these archives were not 
designed for immediate, on-demand 
access to the data. 


•We believe that there is a need to 
bridge the gap between the PDS and 
users who are not “data experts.” 


• Ubiquitous, freely available geo- 
browser platforms are 
technologically well-suited to this 
task, and a natural fit to fill this gap. 



Microsoft 

Worldwide 

Telescope 

(WWT) 


“Google Moon” & 
“Google Mars” 

(in Google Earth 5.0) 



Lunar Imagery 
for Uniview 
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Live Data into Google Earth 

Continuous, automatic data release 


Mission Instrument 
Team Provides 

(via public or private URL) 


Spacecraft Orbit 
Ion, lat, alt, time (CSV file) 


Data Footprint 
lat, Ion of vertices (CSV file) 


Instrument data 
(ISIS cube, PNG, JPEG, etc.) 





publish to 
Google Earth 


KML Convert 


(using ARC Neo-Geography Toolkit) 
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